{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 文档分割"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 一、基于字符分割"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter, CharacterTextSplitter\n",
    "# 块大小\n",
    "chunk_size = 20\n",
    "# 块重叠大小\n",
    "chunk_overlap = 10\n",
    "# 初始化递归字符文本分割器\n",
    "r_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=chunk_size,\n",
    "    chunk_overlap=chunk_overlap\n",
    ")\n",
    "# 初始化字符文本分割器\n",
    "c_splitter = CharacterTextSplitter(\n",
    "    chunk_size=chunk_size,\n",
    "    chunk_overlap=chunk_overlap\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"在AI研究中，由于大模型的规模非常大，参数很多，在大模型上跑完来验证参数好不好训练时间成本很高，所以一般会在小模型上做消融实验来验证哪些改进是有效的再去大模型上做实验。\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['在AI研究中，由于大模型的规模非常大，参',\n",
       " '模型的规模非常大，参数很多，在大模型上跑',\n",
       " '数很多，在大模型上跑完来验证参数好不好训',\n",
       " '完来验证参数好不好训练时间成本很高，所以',\n",
       " '练时间成本很高，所以一般会在小模型上做消',\n",
       " '一般会在小模型上做消融实验来验证哪些改进',\n",
       " '融实验来验证哪些改进是有效的再去大模型上',\n",
       " '是有效的再去大模型上做实验。']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r_splitter.split_text(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['在AI研究中，由于大模型的规模非常大，参数很多，在大模型上跑完来验证参数好不好训练时间成本很高，所以一般会在小模型上做消融实验来验证哪些改进是有效的再去大模型上做实验。']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 默认使用换行符作为分隔符\n",
    "c_splitter.split_text(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 23, which is longer than the specified 20\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['在AI研究中，由于大模型的规模非常大',\n",
       " '参数很多',\n",
       " '在大模型上跑完来验证参数好不好训练时间成本很高',\n",
       " '所以一般会在小模型上做消融实验来验证哪些改进是有效的再去大模型上做实验。']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_splitter = CharacterTextSplitter(\n",
    "    separator='，',\n",
    "    chunk_size=chunk_size,\n",
    "    chunk_overlap=chunk_overlap\n",
    ")\n",
    "c_splitter.split_text(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 长文本分割"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "some_text = \"\"\"\n",
    "在编写文档时，作者将使用文档结构对内容进行分组。 \\\n",
    "这可以向读者传达哪些想法是相关的。 例如，密切相关的想法是在句子中。 \\\n",
    "类似的想法在段落中。 段落构成文档。\\n\\n\\\n",
    "段落通常用一个或两个回车符分隔。 \\\n",
    "回车符是您在该字符串中看到的嵌入的“反斜杠n”。 \\\n",
    "句子末尾有一个句号，但也有一个空格。 \\\n",
    "并且单词之间用空格分隔。\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "148"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(some_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "r_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=80,\n",
    "    chunk_overlap=0,\n",
    "    separators=[\"\\n\\n\",\"\\n\",\" \",\"\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 36, which is longer than the specified 20\n",
      "Created a chunk of size 81, which is longer than the specified 20\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['在编写文档时',\n",
       " '作者将使用文档结构对内容进行分组。这可以向读者传达哪些想法是相关的。例如',\n",
       " '密切相关的想法是在句子中。类似的想法在段落中。段落构成文档。\\n\\n段落通常用一个或两个回车符分隔。回车符是您在该字符串中看到的嵌入的“反斜杠n”。句子末尾有一个句号',\n",
       " '但也有一个空格。并且单词之间用空格分隔。']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_splitter.split_text(some_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['在编写文档时，作者将使用文档结构对内容进行分组。 这可以向读者传达哪些想法是相关的。 例如，密切相关的想法是在句子中。 类似的想法在段落中。 段落构成文档。',\n",
       " '段落通常用一个或两个回车符分隔。 回车符是您在该字符串中看到的嵌入的“反斜杠n”。 句子末尾有一个句号，但也有一个空格。 并且单词之间用空格分隔。']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r_splitter.split_text(some_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 基于Token分割"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import TokenTextSplitter\n",
    "text_splitter = TokenTextSplitter(chunk_size=1,chunk_overlap=0)\n",
    "text = \"foo bar bazzyfoo\"\n",
    "text_splitter.split_text(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 分割Markdown文档"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import MarkdownHeaderTextSplitter\n",
    "\n",
    "markdown_document = \"\"\"\n",
    "# Title\\n\\n\\\n",
    "## 第一章\\n\\n\\\n",
    "李白乘舟江欲行\\n\\n 忽闻岸上踏歌声\\n\\n\\\n",
    "## Section \\n\\n \\\n",
    "桃花潭水深千尺 \\n\\n\n",
    "## 第二章\\n\\n \\\n",
    "不及汪伦送我情\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers_to_split_on = [\n",
    "    (\"#\",\"Header 1\"),\n",
    "    (\"##\",\"Header 2\"),\n",
    "    (\"###\",\"Header 3\")\n",
    "]\n",
    "\n",
    "markdown_splitter = MarkdownHeaderTextSplitter(\n",
    "    headers_to_split_on\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "md_headers_splitters = markdown_splitter.split_text(markdown_document)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'Header 1': 'Title', 'Header 2': '第一章'}, page_content='李白乘舟江欲行  \\n忽闻岸上踏歌声')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md_headers_splitters[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'Header 1': 'Title', 'Header 2': 'Section'}, page_content='桃花潭水深千尺')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md_headers_splitters[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
